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OBJECTIVE 

This Technical Document presents a plan for the implementation of testability into military 
equipment The plan advocates the generation and modification of official MIL-type documents in 
order (1) to establish testability as a design discipline and (2) to result in a family of testability 
documents that can be applied when procuring military hardware. This plan consists of specific 
recommendations in six functional areas comprising testability. 


APPROACH 

The task was accomplished in two parts. The first part entailed grouping existing and proposed 
military standards and design guidance notebooks that relate to testability under the following 
testability headings: 

• Management 

• Technical Requirements 

• Design Guide 

• Documentation 

• Validation 

• Interfaces 

The second part was accomplished by analyzing these groups of documents, noting deficiencies 
and making recommendations to alleviate these deficiencies. In addition, a survey of other 
agencies and commercial companies was performed to determine if progress toward the objective 
was being made elsewhere. 


RESULTS 

Specific recommendations were made in each functional area These recommendations as they 
apply to existing and proposed MIDdocuments are summarized below: 

Proposed New Documents: 

Military Standards 

MIL-STD-ABC Testability Program Requirements 

MII-STD-DEF Testability Demonstration 

- i - 
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MIL-STD-GHI 


Testability Analysis and Report 
MIL-STD-XXXX System Level Performance Monitoring 

Design Guides 
Design for Testability Guide 
Performance Monitoring Design Guide 
Modification - Proposed to Existing Documentation: 

Military Specifications 
MILE-16400 
MILT-24309 
MIL-T-28800 

Other general equipment specifications based on MIL-STD-454 
Military Standards 
MILSTD-415 
MIL STD-454 
MIL STD-490 
MILSTD1521 
MILSTD2076 
MILSTD2077 
MILSTD2111 
Cancellation - Proposed 
MILSTD1326 
MILSTD1345 
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MILSTD1519 




CONCLUSION 


The plan presented herein provides the necessary framework for a program to develop these 
documents as well as for the definition of their content, organization and relationships. This 
framework is sufficient for the structuring of specific tasks in each functional area in order to 
achieve highly testable design in military equipment 

NOTE: Testability is a design characteristic which allows the status (operable, 

inoperable, or degraded) of a unit (system, subsystem, module or component) 
to be confidently determined in a timely fashion. 
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1.0 INTRODUCTION 


The requirement to treat testability as an independent valid design objective within the scope 
of manual and automatic testing, has been recognized only recently. In the past, testability was 
documented by inference within other areas such as maintainability and reliability. As a result, 
many varied testability-related documents currently are in print, in use, and being proposed. These 
documents have been developed over a period of many years and are generally specialized for the 
individual needs of the varied users. This results in a number of differing methods for achieving 
testability in hardware. In some cases these documents are also limited by the state of the tech¬ 
nology at the time they were written. Additionally, varying degrees of conflict, overlap, and dupli¬ 
cation exist between these documents. It is therefore expedient for the testing technology 
community to review the present state of testability-related documentation to determine the best 
approach for using existing documents and for creating new ones. 

This report presents a “systems approach” to the review of testability-related documents. It 
also presents the information which discloses the level of conflict, overlap, and duplication 
between current testability-related documentation. This includes segmentation of the subject into 
major areas of interest Areas that may require changes or the writing of new documents are 
identified and appropriate recommendations are given. The end result is a guide for the develop¬ 
ment of a family of testability documents that can be applied during the procurement phase of 
shipboard hardware, and will aid in the achievement of a high degree of testability and self-test 
features in the hardware. 


2.0 BACKGROUND 

The Testing Technology Office at the Naval Ocean Systems Center (NOSC, Code 921) is 
currently investigating the need for revisions to military standards and other documents that affect 
testing and performance monitoring. This need has been identified as arising from: 

• Testability, as treated in existing documentation, is neither recognized as a design 
discipline in its own right, nor is it explicitly mandated; 

• Program managers having cognizance over equipment development efforts, have no single 
cohesive thread of documentation from which to evoke this design discipline in their 
programs; 

• Some existing documentation does not reflect the latest technology, 

• Potential areas of conflict or incompatibility exist between existing standards and guide¬ 
lines, as well as those being prepared in response to new requirements. 

In the course of the investigation, it was apparent that the application of a “systems approach” 
to the organization of these standards would prove the most effective method of resolving these 
issues. The term “systems approach” simply means to scope out and define the entire problem 
prior to the definition of the solution and to maintain this viewpoint when delimiting each piece of 
the solution. This approach resulted in the present definition for subsequent generation of a 
family of testability-related documents for use by Testability Engineers as well as Program and 
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Acquisition Managers. This family of documents will allow the establishment of testability as a 
design discipline at the same level as the requirements for performance, reliability, maintainability, 
etc. The development of shipboard hardware during the R&D phase is a mix or compromise 
between the various engineering disciplines. In the past, the engineering discipline of maintain¬ 
ability has been the primary method of implementing testability into the hardware. The primary 
proof of testability has been the maintainability demonstration test which occurs during 
qualification testing. The main thrust of this test is aimed more at the mean-time-to-repair 
(MTTR) rather than the testability of the hardware. The MTTR figure can be met using a 
relatively untestable hardware design by relying on a sufficiently complex test scheme. Testability 
impacts the efficiency of testing as well as the MTTR 

Establishing testability as a design discipline will place it at an increased attention level when 
the cost and engineering trade-offs are performed during the R&D phase of the hardware develop¬ 
ment This will increase the testability of military hardware because it is more cost effective and 
practical to design testability into the prime equipment than to develop elaborate test schemes to 
overcome design deficiencies. 


3.0 APPROACH 

A two phased approach was taken. In the first phase, a survey was conducted over a wid^ 
range of military standards, specifications and design guidance notebooks. A total of 23 existing 
and proposed documents were included on the basis of their potential impact on or association 
with testability. Each document was analyzed in the following categories: . 

• Major topics 

• Level of detail 

• Area of coverage 

• Constraints 

• Overlap between documents 

• Contributions toward testability 

Some of the documents surveyed specify requirements in the related fields of maintainability 
and reliability. They were included because they have potential impact on testability. However, 
they also can serve in an instructional capacity due to the fact that they support relatively mature 
but related design disciplines. Thus, the previous experience and example expressed in the 
maintainability and reliability documents could be used in this effort Specifically, the relationship 
of the general equipment standards and specifications was examined with respect to reliability, 
maintainability and related fields. 

The second phase was based on hardware procurement considerations. From the aspect of 
procurement it was found that each document or group of documents support one or more 
functions as follows: 
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• An overall management document for conducting a testability program; 

• A group of documents that addresses the testability technical requirements for hardware; 

• A group of documents, such as a Design Guide, which aid the contractor in implementing 
testability into the hardware; 

• A group of documents that specifies how the testability data is to be prepared and 
presented (Design Review Data, Technical Requirements Data [TRD], Test Programs 
Sets [TPS], etc.); 

• A document that specifies the methods for the validation of testability (demonstration 
tests); 

• A group of documents that specifies the interfaces, data transfer, electrical character¬ 
istics, and sensor selection in support of test/monitoring. 

An analysis was conducted on a function by function basis. Final conclusions were then 
collected and summarized into recommendations. 

Additionally, a survey of “other agencies” and commercial companies was performed to 
ascertain the procurement methods being used to obtain testability of hardware. This was to get an 
indication of the present state of testability relative to current procurement practices. The 
following agencies and commercial companies were surveyed: 

• U.S. Air Force, Air Force Logistics Command, MATE Program; 

• U.S. Army Communications Research and Development Command (CORADCOM); 
Direct Support Automatic Test Support System (DS-ATSS) Program; 

• Boeing Company, Commercial Airplane Division; 

• Lockheed, Commercial Airplane Division; 

• NASA. 

The methods currently being used by the groups surveyed at the time of this report are as 
follows: 

U.S. Air Force, MATE Program 

These Modular Automatic Test Equipment (MATE) Testability guides were not available for 
review because they are competition sensitive. The significant factor is that the subcontractors are 
presenting their methodology for testability to the Air Force, as opposed to the Air Force 
Logistics Command designating the methodology. 
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U.S. Amy, CORADCOM 


The group at the U.S. Army Communications Research and Development Command 
(CORADCOM) do not have testability standards. Instead, they have taken information from an 
independently funded testability design study and developed a testability portion of a hardware 
RFP for “Direct Support Automatic Test Support System (DS-ATSS)”. At the time of this 
report, the submitted proposals had not been evaluated. Testability requirements were placed in 
the performance specification, and data items in accordance with the following DIDs, were 
requested. Appendix C contains these DIDs in their entirety. 

Item DID 

Testability Program Plan DI-A-XXX1 

Testability Analysis Report DI-R-XXX2 

Test Requirements Document DI-T-3734A 

Testability Demonstration Plan DI-T-XXX4 

Except for the Test Requirements Document, the data items are new. 

Boeing 

To achieve testability in hardware, the Boeing Commercial Airplane Division utilizes main¬ 
tainability requirements, implementation of these requirements by the hardware subcontractor, and 
design reviews and testing. They do not have testability-only documents, nor do they treat 
testability as a separate discipline. Testability related requirements are placed in the procurement 
specification. 

Lockheed 

The approach utilized by Lockheed is similar to that used by Boeing. The pertinent factor 
being they do not have a specific “Testability Document”, but use maintainability requirements to 
achieve testability. 

NASA, Space Shuttle, Orbiter 

NASA has taken the position that most of the hardware for the Orbiter portion of the Space 
Shuttle will be repaired by the individual suppliers and testability would not be cost effective. 

They justify this since most of the hardware is low volume and off-the-shelf. 

The “Other Agencies” surveys resulted in the following general conclusions: 

• Testability requirements as a design discipline are not generally specified in current 
acquisition documents; 
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• Maintainability requirements continued to be the most common method of installing 
testability, 

• The maintainability and testability requirements are placed in the “Requirements” section 
of specifications. The inclusion of these maintainability and/or testability requirements 
by the procuring activity without the aid of any associated “Testability Documents” is 
prevalent 

NOTE: An exception to the use of maintainability as a method of achieving testability 
is being attempted by the U.S. Army CORADCOM, DS-ATSS program. 
However, the program has not progressed sufficiently to provide any concrete 
conclusions. 


4.0 RESULTS 

The results of this study are presented by testability function. These testability functions are as 
follows: 


Testability Function 

Paragraph 

Management 

4.1 

Technical Requirements 

4.2 

Design Guide 

4.3 

Documentation 

4.4 

Validation 

4.5 

Interfaces 

4.6 


The Testability Matrix, Figure 1, depicts the testability document grouping by function along 
with deficiencies/recommendations by function. 


4.1 MANAGEMENT 

Testability has not been generally treated as a separate discipline and therefore, except for the 
proposed MID STD-XXX (W. Keiner) “Testability Acquisition Program Requirements for 
Electronic Equipments and Systems”, there are no documents solely dedicated to the overall 
management and validation of testability. Testability as a design discipline is a management 
function. The function of testability management during the procurement phase of development 
requires a MID STD similar to MIDSTD-470 (maintainability) and MIDSTD-785 (for reliability). 
The document “A Study of Testability Standardization for Electronic Systems and Equipment” 
(W. Keiner), contains the testability management elements with which to develop a testability 
management document, MIDSTD-ABC (Proposed). “Testability Acquisition requirements for 
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TESTABILITY 


DOCUMENT 

^^^njNCTIO^ 

MANAGEMENT 


TECHNICAL 

REQUIREMENTS 


DESIGN GUIDE 


MIL-STD-ABC (Proposed) 


MIL-STD-XXX (Proposed) W. Kemer 

MIL-E-16400 

Mll-T-28800 

MIL-STD-454 

MIL-STD-41 5 


Built-In Test (BIT) Design Guide 
Operational Monitoring (ORMS) Design 
Guide (Proposed) 

Design for Testability Guide. W Kemer 
(Proposed) 

MIL-STD-1390 
MIL-STD-1 364 


MIL-STD-151 9 
MIL-STD-1 345 
MIL-STD-2076 
MIL-STD-2077 
D-790 

MIL-T-24309 
MIL-STD-21 1 1 
MIL-STD-24255 
MIL-STD-490 
M1L-STD-GH1 (Proposed) 
MIL-T-24424 
MIL-STD-31 10 
MIL-STD-1 521 


MIL-STD-471 
MIL-STD-DEF 

Demonstration (Proposed) Testability 


MIL-STD-XXXX (Proposed) NOSC) 

MIL-STD-1657 
SDMS (I/O Modules) 

MIL-STD-1553 
MIL-STD-1397 
DO D-STD-1399 
IEEE-STD-488 

MIL-STD-X. On-Line/Off-Line 
Interface (Proposed) 

MIL-STD-1326 


Testability Program 


Testability Reqmts 
Equipment Spec. 
Equipment Spec. 
General STD. 
Testability Reqmts. 



Released Design Guide 
(Proposed) 




A document does not exist in Testability as a discipline Need an operational moni- 
released form to manage a does not exist in equipment toring and design for test- 

testability program by an specs or general standards. ability guide to bridge 

R&D contractor. educational gap. 


Utilize W. Keiner study to Change the equipment Draft an operational moni- 

develop a testability program specifications and MIL-STD’s toring design guide. Convert 

document along with sup- to include testability as a the W. Keiner testability 

porting DID'a. design discipline. course to a testability design 

guide. 



Figure 1. Testability Matrix 

























DESIGN GUIDE 


DOCUMENTATION 


VALIDATION 


INTERFACES 


Released Design Guide 
Proposed) 

i Proposed) 

Level of Repair 
GPETE 


TRD 

TRD 

TRD 

TPS 

TPS 

TSP 

TRS 

TRS 

Specification Practices 
Test. Anal. Et Report 
TRS 
ROR 

DSN Reviews/Audits 


Maintainability Test 
Testability Demonstration 
(Proposed) 


System Level Monitoring 
(Proposed) 

Physical Interface 
Equipment Spec. 

Series Data Bus 
Computer Interfaces 
Shipboard Interface 
Parallel Data Bus 
(Proposed) 


Need an operational moni¬ 
toring and design for test¬ 
ability guide to bridge 
educational gap. 


Testability analysis docu¬ 
ment not released. Duplica¬ 
tion in TRD area. Testability 
test document not released. 
TRD and TPS documents not 
universal. 


MIL-STD-471 does not place 
priority on how well the test 
equipment works, but how 
long it operates. 


On-Line Testability 


Interfaces for shipboard hard¬ 
ware, both on-line and off¬ 
line have not been 
standardized. 


Draft an operational moni¬ 
toring design guide. Convert 
the W. Keiner testability 
course to a testability design 
guide. 


Convert a portion of W. 
Keiner study to a testability 
analysis document. Develop 
a universal TRD and TPS 
document utilizing MIL- 
STD's 2076 and 2077 as a 
base. 


Convert a portion of the 
W. Keiner study to a testabil¬ 
ity MIL-STD for validation. 


Standardize the interface 
for both on-line and off-line 
shipboard hardware. 


Figure I. Testability Matrix 
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Electronic Equipment and Systems" (MIL STD-XXX, Proposed) will also assist greatly in 
testability management The proposed allocation of technical data from these reports is presented 
in the Technical Requirements section of this report The establishment of testability as a design 
discipline and the organizational role of the testability engineer are examined in the material to 
follow. 


4.1.1 TESTABILITY ENGINEERING 

The discipline of testability emerged primarily thro.igh the maintainability function. Therefore, 
consideration must be given to maintainability and similar disciplines to review the pitfalls that 
accompanied their emergence as entities. Similar difficulties for testability can then be avoided. 

Historically, in engineering companies, maintainability has been a support function, not part of 
the performance design engineering team. As such, it has been staffed under an engineering 
support or quality assurance group along with reliability. This tends to give the discipline a 
classical watchdog (quality assurance role) connotation which tends to form a barrier between the 
performance design engineering team and the maintainability/reliability engineers. The maintain¬ 
ability/reliability functions achieve the most efficient implementation when a working group 
relationship is established with the performance design engineering group. 

To date, the implementation of testability into the hardware has not been a high priority item 
even within the scope of a maintainability demonstration. During such demonstrations, the 
contractor usually supplies (and thus controls) the list of faults along with their implementation, 
and thus assures the successful passing of the test despite the presence of any testability 
shortcomings. Classically the implementation of testability has been handled by assigning the 
design effort to the same engineer that had the design responsibility for performance. This usually 
causes a priority and time scheduling problem that results in testability being relegated to a “when 
time permits” situation. Also contributing to the problem is the fact that the performance design 
engineer does not normally have the experience or background in testability-related disciplines 
(ATE, ATLAS, TPS, etc.) which are instrumental in achieving a testable design. 

In any consideration of adding the testability function to the duties of maintenance or 
reliability engineers, the following considerations must be made: 

• Maintainability engineers tend to be involved in the “packaging” of the hardware by 
function, performing maintainability predictions, and in the maintainability demonstration 
tests, along with monitoring the design; 

• The reliability engineer is involved with parts procurement, parts application, reliability 
prediction, failures, and the reliability demonstration tests, along with monitoring the 
design; 

• Because of the function of testability, (i.e., the confident timely determination of 
equipment status), the testability engineer must be more hardware design oriented than 
the maintainability and reliability engineer. 





Due to the differences in the design disciplines of testability as compared with maintainability and 
reliability, the testability function is optimized by assigning the testability engineer as part of the 
performance design engineering team. The enhancement of the working relationship between the 
testability engineer and performance design engineers should increase the application of testability 
into the design process. 


4.1.2 TESTABILITY ENGINEERING EFFECTIVENESS 

Effective development of hardware during the design phase is a composite of all engineering 
disciplines which results in a cost-effective design. All design disciplines (including testability) have 
to be designed in, not tested in after the fact The establishment of testability as a design discipline 
through testability requirements, Contract Data Requirements List (CDRL) items, and a 
testability demonstration test will result in more testable hardware. 

The effectiveness of the testability engineering role will depend on the following: 

• A close working relationship with the performance design engineering team; 

• A testability demonstration test as part of the qualification tests. This places importance 
on testability as a design discipline, an attention getter. Testability test results should also 
be considered as Design Review, Audit (FCA/PCA) requirements; 

• A clearly defined testability requirement in the performance specification. This avoids 
ambiguities when design trade-offs are performed which affect performance, testability, 
maintainability, reliability, safety, etc.; 

• Status as a design discipline equal to performance, maintainability, reliability, safety, etc. 
This is achieved by program requirements which are attention getters for the testability 
discipline. These program requirements are as follows: 

- Testability Program Plan; 

- Testability Analysis; 

- A subject for the Preliminary Design Review (PDR) and Critical Design Review (CDR) 
as specified in the performance specification, proposed testability requirements, 
MIL-STDs and CDRLs; 

- Testability demonstration test as part of the qualification tests. 


4.2 TECHNICAL REQUIREMENTS 

Technical requirements pertain to those testability conditions that are specified to a contractor 
for the purpose of delineating desired limits on the hardware design. These testability requirements 
are in MIL-equipment specifications, MIL-STDs and the hardware performance specification. 
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The hardware performance specification (per MILSTD-490) is the starting place for the analysis 
herein, followed by the MILequipment specifications and testability MIL-STDs. 

The following documents, as shown in Figure 1, are grouped under the Technical requirements 
function: 

• MILSTD-XXX Proposed, W. Keiner, Testability Acquisition Program Requirements 
for Electronic Equipments and Systems; 

• MILE-16400, Electronic Equipment, Naval Ship and Shore: General Specification; 

• MILT-28800, Test Equipment for use with Electrical and Electronic Equipment, 
General Specifications for, 

• MILSTD-454, Standard General Requirements for Electronic Equipment; 

• MILSTD-415, Test Provisions for Electronic Systems and Associated Equipment, 
Design Criteria for. 


4.2.1 HARDWARE PERFORMANCE SPECIFICATIONS 

These hardware performance specifications are drafted to the applicable appendix of MIL 
STD-490, “Specification Practices". This standard does not currently make provisions for 
testability as a specification item, as it does for maintainability and reliability. MILSTD-490 is 
discussed here only in the context that testability requirements are placed in the hardware perfor¬ 
mance specification. MILSTD-490 is grouped in the Testability Matrix, Figure 1, under the 
documentation function. 


4.2.2 APPLICATION OF REQUIREMENTS 

The application of a given program design discipline to a family of military equipment 
specifications is dependent upon the ability to pass requirements associated with that particular 
discipline from general documents to detailed equipment specifications. The requirement of main¬ 
tainability, as applicable to electronic equipment design, is used in the following example since it 
is a long acknowledged design discipline. 


4.2.2.1 Maintainability Example 

General specifications, such as MILE-16400 for electronic equipment and MILT-28800 for 
test equipment, specify requirements for maintainability (as well as other acknowledged program 
disciplines). These requirements are consolidated and standardized in MILSTD-454, “Standard 
General Requirements for Electronic Equipment". Certain of these requirements complement 
each other in application to equipment design and can be categorized as hardware and program 
(design discipline) requirements as follows. 
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MI D STD-454 

Design Discipline Requirement Hardware Requirement 

Req. 54 - Maintainability Req. 7 - Interchangeability 

Req. 36 - Accessibility 

It can be seen that the subjects of “interchangeability” and “accessibility” are maintainability 
hardware requirements. 

Having determined the applicable requirements (still using Maintainability as an example), the 
functions of each along with applicable documents, are specified in the general standard (MID 
STD-454) as follows: 

• Maintainability 

Maintainability Program MID STD-470 

Maintainability Prediction MIL-HDBK-472 

Maintainability Verification MIDSTD-471 

• Accessibility 

Definition of Item Levels, Item Exchange¬ 
ability Models, Related Terms, etc. MID STD-280 

This hierarchy of documents for maintainability is depicted in Figure 2. 


4.2.2.2 Proposed Application for Testability 

The application of testability to a family of specifications necessitates that applicable 
requirements be passed or traced from specific testability-oriented documents to general 
ones. As with maintainability, the starting point for examining this flow is the equipment 
level specifications (MIDE-16400, MIDT-28800, etc.), followed by a general requirements 
document (MIDSTD-454) that specifies hardware and program (design discipline) requirements 
and documents. A problem becomes apparent in the application of testability as a design 
discipline, due to the appearance of voids in testability requirements and documentation. For 
example, the equipment specifications (MIDE-16400 and MIDT-28800), and the general 
standard (MIDSTD-454) do not presently contain testability program (design discipline) 
requirements. Therefore, modification of MIDE-16400 and MIDT-28800 (et al) is necessary to 
draw these requirements from testability documents. Similarly MIDSTD-454 will need 
modification to include testability as a program requirement which will specify a family of 
testability program functions and documents as follows: 














MIL-STD-454 


Design Discipline Requirement Hardware Requirement 

Requirement XX Testability (Proposed) Req. 32 -Test Provisions 

Req. YY - Interfaces (Proposed) 

It should be noted that Requirement 32 “Test Provisions” contained in MIL-STD-454 is a 
testability hardware requirement This requirement needs to be updated to include interface require¬ 
ments and/or add a new MIL-STD-454 Requirement YY, Interfaces. This allows for testability 
hardware and program requirements in MID STD-454 that may be applied in parallel as with maintain¬ 
ability requirements. The W. Keiner study, “A Study of Testability Standardization for Electronic 
Systems and Equipment”, contains the testability counterparts to maintainability. The data contained 
in the proposed W. Keiner document could be used to develop a family of testability documents by 
function similar to the family of maintainability documents. This family of MIL-STD-454 specified 
testability program documents is proposed as follows: 

• Testability 

Testability Program Requirements 

Testability Demonstration 

Testability Analysis and Report 

• Test Provisions 

Test Provisions for Electronic Systems and 
Associated Equipment, Design Criteria for MID STD-415 

• Interfaces 

TBD TBD 

The proposed hierarchy of testability documents is illustrated in Figure 3. 


MID STD-ABC (Proposed) 
MIDSTD-DEF (Proposed) 
MIDSTD-GHI (Proposed) 


4.3 DESIGN GUIDE FUNCTION 

The establishment of testability as a design discipline will require more than specifying requirements 
in MIDSTDs, Statements of Work( SOWs) and performance specifications. The design engineers that 
are involved in the performance aspects of the hardware do not normally have background or 
experience in testability. The design engineers that have testability experience are usually involved in 
designing test equipment, not operational hardware. 


- 14- 


MIL-E-16400 MIL-T-28800 

EQUIPMENT Electronic Equipment. Test Equipment for use with MIL-X (Others) 

SPECIFICATIONS Naval Ship and Shore Electrical and Electronic 

General Specification Equipment, General Spec. 
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Figure 3. Testability Hierarchy of Documents (Proposed) 











An educational process will naturally occur when testability is implemented as a design discipline. 
This educational process will either occur through trial and error or through the use of planned 
documentation. The use of testability design guides will help bridge this potential educational gap. 


4.3.1 BUILT-IN-TEST DESIGN GUIDE (BIT) 

The Built-In-Test(BIT) Design Guide is currently available for use in hardware procurement and 
may be updated and expanded as new techniques and electronic components become available. 
Although the BIT Design Guide discusses operational monitoring, it is primarily associated with 
Go/No-Go testing. The classical Go/No-Go testing concept is normally based on reporting that the 
hardware is either operational or non-operational. 

Early BIT efforts were developed to meet the maintenance concept; that is, how to determine if the 
equipment is ready to operate and how to isolate a failure. How well it was working was not a driving 
consideration. 


4.3.2 OPERATIONAL MONITORING (ORMS) DESIGN GUIDE (PROPOSED) 

Operational monitoring (part of ORMS concept) is involved with how well the equipment is 
operating. This is required to assess the degree of performance and/or degraded performance capability 
of shipboard hardware. The current BIT Design Guide does not stress methods of implementing 
operational monitoring. The addition of special “Operational Monitoring” sections or the develop¬ 
ment of a separate “Operational Monitoring Design Guide” will help in the testability educational 
process. 


4.3.3 DESIGN FOR TESTABILITY GUIDE 

The conversion of the W. Keiner “Design for Testability Course” to a written design guide would 
be a beneficial primary document for the testability educational process. The BIT and the proposed 
ORMS design guides would be related specialized documents. 


4.3.4 MIDSTD-1390, LEVEL OF REPAIR 

MIL-STD1390, Level of Repair (LOR), is placed in the same category as the design guides 
because of its value in performing testability design trade-off studies by the hardware development 
contractors. Normally the LOR is used during the acquisition phase in determining the maintenance 
concept During the design of the hardware, after the maintenance concept has been established, 
the LOR may be used to perform those hardware design/cost trade-off studies that affect 
testability. 
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4.3.5 MIDSTD-1364, STANDARD GENERAL PURPOSE ELECTRONIC TEST 
EQUIPMENT 

This MID STD for general purpose electronic test equipment (GPETE) was placed under the 
design guide function because the selection of GPETE can be recommended, but not dictated by 
contract The GPETE listed in MIDSTD-1364 would be a “first-advice” shopping list for use by 
the hardware development contractor. This document and the preceding cited and proposed guide- 
type documents would be discussed in the statement of work and supplied as part of the Request 
for Proposal (RFP) package. They would not be invoked contractually, but would be supplied as 
“information only” documents to be used by the contractor as he sees fit 

4.4 DOCUMENTATION 

The following documents, as shown in Figure 1, are grouped under the Documentation 
function: 


• MIL-STD-1519, Test Requirements Document preparation of; 

• MIL-STD-1345, Data, Measurement in Support of Maintenance, Calibration, and 
Repair of Electronic Equipment 

• MIL-STD-2076, UUT Compatibility with ATE, General Requirements for, 

• MID STD-207 7, Test Program Sets, General Requirements for, 

• D-790, Test Program Sets, General Requirements for, NESEC SD STD; 

• MIDT-24309, Technical Support Plan for Electronic Equipment 

• MIDSTD 2111, Technical Repair Standards (4G Repairables), Preparation of (Draft); 

• MIDT-24255, Technical Repair Standards, Submarines; 

• MIDSTD-490, Specification Practices; 

• MIDSTD-GHI (Proposed), Testability Analysis and Report, Preparation of, 

• MIDT-24424, Technical/Maintenance Overhaul and Repair Standards; 

• MIDSTD-3110, Restoration, Overhaul, and Repair of Electronic Equipment 

• MIDSTD-1521, Technical Reviews and Audits for Systems, Equipments, and 
Computer Programs. 

4.4.1 DOCUMENTATION FUNCTION 

The testability documentation function relates to those documents that require the drafting or 
submittal of testability related data. This data may be required at Preliminary Design Reviews 
(PDR), Critical Design Reviews (CDR), and audits; in addition to being cited by Contract Data 
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Requirements List (CDRL) item required by contract The types of testability data are as follows: 

• Performance Specification (MIDSTD-490); 

• Testability Analysis and Report (Proposed MIDSTD-GHI); 

• Test Requirements Data (TRD); 

• Testability Demonstration (Proposed MIL-STD-DEF); 

• Technical Support Plan (TSP); 

• Test Program Sets (TPS); 

• Technical Repair Standards (TRS); 

• Restoration, Overhaul, and Repair (ROR); 

• Technical Reviews and Audits. 

The flow of testability related data will generally follow the top-to-bottom sequence depicted in 
Figure 4, Testability Documentation Tree. A discussion of these testability data items is presented 
in the following sections. A more detailed analysis of selected documents pertaining to 
documentation is contained in Appendix A, Document Survey. 

4.4.1.1 Performance Specification 

The performance specifications are formatted per MIL-STD-490. MID STD-490 is placed 
under the documentation section (para 4.4) for this report because these specifications, depending 
on production volume, eventually become hardware equipment specifications. MID STD-490 was 
also referenced under the Technical Requirements section (para. 4.2) because the original RFP 
performance specifications contain testability design requirements. MIDSTD-490 does not 
currently contain specific testability sections as it does for reliability and maintainability. This is 
required to establish testability as a design discipline. See Section 4.2, Hardware Performance 
Specifications. 

4.4.1.2 Testability Analysis and Report 

The Testability Analysis and Report (MIDSTD-GHI, Proposed) is a proposed fall-out from 
the W. Keiner report, “A Study of Testability Standardization for Electronic Systems and 
Equipment”. This analysis will provide qualitative and quantitative data that determine the 
equipment’s ability (or lack of ability) to pass a testability demonstration test This MID STD will 
contain provisions for a Testability Analysis Report (TAR) for documenting results of the 
testability analysis. The TAR will be presented at PDR/CDR, audits, and will be delivered per 
CDRL item instructions. This testability analysis is in addition to the maintainability analysis. 

The prime concern will be the determination that the quantitative changes (deltas) are detectable 
at the interfaces following a failure. This may be implemented as a modified Failure Modes and 
Effect Analysis (FMEA) currently in use by reliability engineers. The modified portion of the 
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FMEA would be the quantitative changes at the interface in addition to the effect of the failure. 
These quantitative changes (deltas) would then be used to determine the tolerances and accuracy 
requirements of associated sensing equipment This is the lead-in to Test Requirements Data 
(TRD) and the Testability Demonstration Plan (proposed). 

4.4.1.3 Test Requirements Data (TRD) 

The TRD will utilize data from the TAR The TRD is currently covered in the following MIL 
documents: 

• MIDSTD-1519, Test Requirements Document Preparation of; 

• MIDSTD-1345, Data, Measurement in Support of Maintenance, Calibration, and 
Repair of Electronic Equipment 

• MIDSTD-2076. Unit Under Test Compatibility with ATE, General Requirements for. 

A discussion of these MIL-STDs follows: 

MIDSTD-1519 

MILSTD-1519 performs the same general function as Appendices A, C and D of MID STD- 
2076. MILSTD-1519 is not set up to use ATLAS, whereas MID STD-2076 does. 

MIDSTD-1345 

MIDSTD-1345 works with MIDSTD-2111, “Technical Repair Standards (4G Repairables), 
Preparation of’. MIDSTD-1345 is primarily a data document for the preparation and submission 
of data. This document is primarily associated with GPETE, not ATE. 

Requirements for test procedures are specified as (a) performance test procedures, (b) fault 
location procedures, and (c) alignment procedures. 

a. Performance test procedures are typified by those procedures developed by an equipment 
contractor for qualification and acceptance testing of the operational hardware. Normally 
this testing is not performed by ATE. 

b. Fault location procedures are typified by those procedures used in, or in conjunction with, 
a Technical Repair Manual. Reference is not made relative to ATE which requires TPS 
(Example: MIDSTD-2077). 

c. Alignment procedures are typified by those procedures used in a depot operation 
(Technical Repair Manual/Calibration Procedure). 

In general, MIDSTD-1345 is not specifically set up for ATE. Data is specified, but not 
specifically for the generation of TPS. 


- 19 - 



MIL-STD-2076 


MIDSTD-2076 was set up to work with MIDSTD-2077, Test Program Sets, and AR-10, 
(MIL-STD-2084, Proposed), "Maintainability of Avionics Equipment and Systems, General 
Requirements for”. This MIUSTD used the terms “Avionics" and “WRA” which are not 
suitable for shipboard application. These avionic terms would need to be standardized to more 
general terms to make MIUSTD-2076 a more universal document 

Section 5.1.1, Design for Test on ATE. The contents of this section are not requirements, but 
rather recommendations and examples, and are more applicable in the “Testability Design 
Guide” (Proposed). 

Section 5.1.2, Test Points, apply to, and should be inserted in MIDSTD-415, “Test 
Provisions for Electronic Systems and Associated Equipment Design Criteria for”. 

Section 6.1.2, UUT Documentation Analysis and Hardware Inspection, are more functional in 
the following proposed documents: 

• MIDSTD-GHI, Testability Analysis and Report 

• MIUSTD-DEF, Testability Demonstration 

NOTE: See Management and Technical Requirements sections of this study. 

Appendix A, Test Requirements Data, with companion Appendices C and D, perform the 
same general function as MIL-STD-1519, “Test Requirements Documentation, Preparation of’. 

Appendix B, Compatibility Task Scoring Data: A scoring or rating system is more functional 
in the proposed MIDSTD-GHI, “Testability Analysis and Report”. This type of analysis, if 
performed early in the R&D cycle (prior to PDR), can influence the design. If performed at or 
after the qualification testing, its function becomes passive. 

Appendices C and D are as described in Appendix A. 

Appendix E, “Application of UUT Failure Rate Data”: This type of data can influence the 
design of the hardware and should be available prior to the PDR This type of data is also more 
functional in the proposed MIL-STD-GHI, “Testability Analysis and Report”. 

Appendix F, “Fault Isolation (FI) Test Requirement, Fault Isolation Data”: This appendix 
performs the same general function as MIL-STD-1345. The major differences are the use of 
functional and data flow charts (ATLAS Test Procedures) in MIL-STD-2076, while MIDSTD- 
1345 calls out detailed test procedures. 

The TRD will be the data base for the following documents: 

• Testability Demonstration Procedure 

• Technical Support Plan (TSP) 
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• Test Program Sets (TPS) 

4.4.1.4 Testability Demonstration (MIL-STD-DEF) Proposed 

The content for Testability Demonstration Plan/Procedure/Report documents are contained in 
the W. Keiner report “A Study of Testability Standardization for Electronic Systems and 
Equipment”. The testability demonstration plan, procedure and report documents will be CDRL 
items that will document the testability demonstration test 

Testability Demonstration Plan 

The Testability Demonstration Plan delineates the test requirements and the general 
implementation of the testability testing This allows for a meeting of the minds, between the 
procuring agency and the contractor, concerning the interpretation of the test requirements and 
their implementation. The Testability Demonstration Plan is expected to be drafted after the TAR 
and prior to the TRD (see Figure 4). The plan will be submitted for approval prior to the 
testability demonstration procedure. 

Testability Demonstration Procedure 

The Testability Demonstration Procedure will follow the TRD (see Figure 4). The procedure 
will specify the method of verifying the test requirements data. 

Testability Demonstration Report 

The Testability Demonstration Report follows the Testability Demonstration Test 


4.4.1.5 Technical Support Plan (TSP) 

The TSP, MIL-T-24309, is an accumulation of all data that are pertinent to the support of 
equipment Included in addition to testability data are: 

« Maintenance concept; 

• Plan for maintenance; 

• Reliability design data; 

• Maintainability design data. 

The testability related data are specified as (1) modular construction and assembly design data, and 
(2) test point and test equipment data. The types of data specified in MIL-T-24309 are normally 
available as CDRL items separately, but not in one place. In the event MIDT-24309 is to be 
specified, the following recommendation is submitted in the testability area: 

• Modify paragraph 3.2.1.5, “Modular Construction and Assembly Design Data” and 

3.2.1.6, “Test and Test Facilities Data”, to reference information contained in the Test 
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Requirements Data (TRD) as specified in MIDSTD-1519, MIL-STD-1345, or MID 
STD-2076. 


4.4.1.6 Test Program Sets (TPS) 

The TPS documents available to the reviewer were MID STD-2077 and D-790. MIDSTD- 
2077 is a NAVAIR release while D-790 is an internal document prepared by NESEC. Generally 
speaking, the D-790 standard closely parallels MIDSTD-2077. Major differences are: 

• D-790 terminology is more universal. For example, it does not use restrictive terms such 
as “avionics” and “NAVAIR”. It tends to avoid use of MIDdocuments in specifying 
requirements and is, therefore, ambiguous; 

• MIDSTD-2077 is more definitive than D-790. It makes provisions for validation of the 
TPS by the Government agency, whereas D-790 is weak in this area. 

Neither document is directly applicable to shipboard equipment MIDSTD-2077 would have to 
be made less restrictive by broadening terminology beyond the scope of typical NAVAIR 
terminology, and D-790 would have to be more definitive in the area of requirements to be 
considered for status as a released MIDSTD. 

NOTES: 1. MIDSTD-2077 is a companion document to MIDSTD-2076 as indicated by 
paragraph 5.10.1.1, “Program Design Data (PDD)” of MIDSTD-2077. 

2. It is important to assure the software (TPS) is available in time to perform the 
testability demonstration. This potential time phasing problem will be made an 
item to be included in the testability demonstration plan. An ideal situation is 
to have the hardware design contractor develop the TPS to match the Govern¬ 
ment agency’s test requirements. 

Detailed comparisons of MIDSTD-2077 and D-790 are contained in Appendix A, Document 
Review. 


4.4.1.7 Technical Repair Standards (TRS) 

The TRSs are covered by the following documents: 

• MIDSTD-2111, Technical Repair Standards (4G Repairables), Preparation oft 

• MIDT-24255, Technical Repair Standards (For Submarines Only); 

• MIDT-24424, Technical/Maintenance Overhaul and Repair Standards. 

The discussion of these documents follows. 

MIDSTD-2111 

MIDSTD-2111 covers repair at the depot level of maintenance. This document was set up to 
work with MIDSTD-1345 and MIDSTD-1364. MIDSTD-2111 does not make specific 
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provisions for ATE and the supporting TPSs. MIL-STD-1345 is a TRD document, but not for 
ATE. The addition of an ATE section which references MILSTD-2076 (TRD) and MILSTD- 
2077 (TPS) would augment MILSTD-2111. A more efficient approach would be to cancel MIL 
STD-1345 and use MILSTD-2076 for the TRD data, for the use of ATE and GPETE. This is 
based on the following: 

• Appendix E of MILSTD-2076 covers the non-ATE testing in terms of ATLAS in lieu 
of detailed test procedures; 

• The detailed test procedures of MILSTD-I345 call for specific test equipment which has 
the following limitations: 

— Identification of test equipment by model number (vendor) is limited and shows 
preference by manufacture. 

— A problem of substitution results in when the specific test equipment is not available 
at the depot in question. 

- The supporting TRD (ATLAS) will not be available in event ATE is to be used 
later. 

MILT-24255 

MILT-24255 was drafted for the overhaul of submarines, and does not apply directly to 
repair of hardware at a depot facility. 

MILT-24424 

MILT-24424 was drafted for the overhaul of ships. This document is similar to MILT- 
24255. 


4.4.1.8 Restoration, Overhaul, and Repair (ROR) 

MILSTD-3110, Restoration, Overhaul, and Repair of Electronic Equipment, is listed under 
documentation because testing is a part of it In the tier of documents, MILSTD-3110 uses TRSs 
which are covered in MILSTD-2111, MILT-24255, and MILT-24424. 


4.4.1.9 General Purpose Electronic Test Equipment (GPETE) 

The general purpose electronic test equipment (GPETE) is shown in Figure 4, Testability 
Documentation Tree, for those cases where ATE is not used. The GPETE data is specified in the 
TRD and used in the TRS and ROR documents. The TRS/ROR documents utilize the TRD, 
GPETE and TPS data. 
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Figure 4. Testability Documentation Tree 
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4.4.1.10 Design Reviews and Audits 

MIL-STD-1521 checklists contain maintainability, reliability and other data to be covered 
during design reviews and audits. No such provisions exist for testability. 


4.5 VALIDATION 

The following documents are grouped under validation, as shown in the Testability Matrix, 
Figure 1: 

• MIDSTD-471, Maintainability Verification/Demonstration/Evaluation 

• MIL-STD-DEF (Proposed), Testability Demonstration 

Testability validation is performed as part of hardware qualification and acceptance testing. 
This also includes testability software. The purpose of these tests is to prove that the hardware 
design meets the testability requirements. Classically, the testability validation has been 
approached through the maintainability demonstration test per MIL-STD-471. The maintain¬ 
ability demonstration ruling parameter is the mean-time-to-repair (MTTR). The MTTR parameter 
can be reduced to the following time steps: 

Function Using 

Detect Test equipment or BIT 

Isolate Test equipment or BIT 

Remove/Replace Technician 

Verify (retest) Test equipment or BIT 

The faster the test equipment or BIT operates, the more time is allowed to remove/replace. The 
total MTTR time is the sum of the individual function times. MIDSTD-471 restricts the time in 
which the test equipment must perform. This is to the direct detriment of the important testability 
factor of how well it performs. 

Also, the selection of maintenance tasks for the maintainability demonstration is based on a 
stratification procedure; the intent being to ‘‘(a) allow for the selection of maintenance tasks in 
such a manner that the selection simulates the failure frequency of the test unit in actual operation 
(units with low MTBFs will be selected more frequently than units with higher MTBFs), and (b) 
ensure that a proportionately representative sample of task types/times are selected." A typical 
test is based on 50 tasks which are selected from 200 tasks. The 200 tasks along with the method 
of implementation are normally selected and submitted by the contractor. The customer will select 
50 tasks from the 200. Since the contractor selects the 200 tasks, there is little chance that his 
hardware will fail the maintainability demonstration. With the contractor controlling the outcome 
of the maintainability demonstration and the controlling factor being the MTTR parameter, a 
question exists as to how testable the hardware design really is. The information contained in the 
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W. Keiner study, when formatted into a testability demonstration MIl^STD, would overcome the 
shortcomings of MIDSTD-471. This is not a recommendation to scrap MIL-STD-471, but to 
augment the maintainability demonstration with a testability demonstration either concurrently or 
at a different time. 


4,6 INTERFACES 

The following documents are grouped under the interface function in the Testability Matrix, 
Figure 1: 

• MIDSTD-XXXX (NOSC Proposed), System Level Performance Monitoring, General 
Requirements for, 

• MIL-STD-1657, Switch Equipment, Combat System, Command and Control, and Fire 
Control, Requirements for, 

• Ships Data Multiplexer System (SDMS) (I/O Modules), Equipment Specification; 

• MIDSTD-1553, Aircraft Internal Command/Response Time Division Multiplex Data 
Bus; 

• MIL-STD-1397, Input/Output Interfaces, Standard Digital Data, Navy Systems; 

• DOD-STD-1399, Interface Standard for Shipboard Systems; 

• IEEE STD 488, 1978 Standard Digital Interface for Programmable Instrumentation; 

• On-Line/Off-Line Interface (Proposed); 

• MIL-STD-1326, Test Points, Test Point Selection and Interface Requirements for 
Equipment Monitored by Shipboard On-Line Automatic Test Equipment 

Although this list of interface documents is not complete, it is an indication of the various 
documents that have been developed in an attempt to standardize interfaces. The hardware inter¬ 
face can be categorized as physical, performance, and contractual. The physical interfaces are 
determined by dimension, connector callout, pin designation, fluid coupling, etc. The performance 
interfaces comprise electrical, mechanical and information format and content Contractual 
interfaces are physical and performance interfaces that can be controlled and checked by 
qualification and acceptance testing The hardware interfaces (physical and performance) are 
contractual During acceptance testing the information available for determining compliance is 
normally at the surface of the package and through connectors and/or couplings. Because of this 
contractor-customer contractual interface, the standardization of interfaces for specific types of 
hardware will result in reduced life cycle costs. 
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The selection of an interface for use between equipment and a data bus (on-line and off-line) is 
based on the following: 

• Performance - 

— Data rates determine the type of bus. Parallel buses are faster while serial buses are 
slower but better over long distances. 

— Information content across the interface will determine the word length requirements. 
-- Electrical interfaces are controlled by the bus I/O modules. 

• Physical - 

- Physical interfaces are controlled by the bus I/O modules. 

• Contractual 

— Contractual interfaces are those (performance or physical) that can be controlled and 
checked by qualification and acceptance testing. 

Because of the varied performance requirements, no one interface will meet the requirements 
of all users. A standard shopping list of interfaces should be available for selection to meet the 
user’s performance requirements. DOD-STD-1399, being a general standard, and MIL-STD- 
XXXX (NOSC Proposed) are the logical places to place these shopping lists. This shopping list 
would reference the other data bus interface standards by type and use. 

MIL-STD-XXXX (NOSC) 

MIL-STD-XXXX (NOSC Proposed) presents a methodology of developing a set of sections 
or MIL-STDS that are devoted to the standardization of performance monitoring for classes of 
hardware. Three levels of interface are proposed to allow flexibility in meeting the user’s tolerance 
requirements. 

MIL-STD-1657 

MIL-STD-1657 is primarily a physical interface document that is referenced by MIDS- 
17000. This is applicable to combat systems. 

Ships Data Multiplexer System (SDMS) Equipment Specification 

The SDMS I/O modules standardize the interface between the SDMS and shipboard 
equipment This will be applicable to ships that will have the SDMS system. 

MILr STD-1553 

MIL-STD-1553 specifies a type of series data bus. This was developed primarily to reduce the 
amount of hardwiring in aircraft 
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MIL-STD-1397 


MIUSTD-1397 is an interface document primarily concerned with computers and the 
following digital bus interfaces: 

• Type A (NTDS slow); 

• Type B (NTDS fast); 

• Type C (ANEW); 

• Serial. 

DOD-STD-1399 

DOD-STD-1399 with supporting sections specifies interfaces for shipboard equipment These 
interfaces include the following types: 

• Support Services (Environmental): 

• Electronic Information (Functional); 

• Controlled Factors (Environmental); 

• Weapons Control (Functional); 

• Uncontrolled Environment (Environmental); 

• Operational Factors (Environmental); 

• Electrical Information (Functional); 

• Mechanical Information (Functional). 

The physical types of interfaces are specified under “controlled factors (environmental)”. An 
example of this is Digital Computer Grounding (Section 406, MIL-STD-1399) where MIL-C-915 
cable is specified This is similar to the approach used by MIL-STD-1657 which specifies 
physical parameters, but is a subtier document of MIUS-17000. MIL-S-17000 is an equipment 
specification whereas Section 406 is a subset of a MIUStandard (MIUSTD-1399). 

IEEE STD-488 

IEEE STD-488 is a parallel digital interface bus that was developed for a family of 
programmable test equipment 

On-Line/OfF-Line Interface (Proposed) 

The establishment of a standard on-line/off-line interface would apply to new designs. This 
would require a means to balance performance monitoring with classical BIT. The differences 
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between performance monitoring and the classical BIT are well explained in Section 3.2.7 
(Tolerances) of the Built-In-Test Design Guide, June 1980. Classical BIT is more concerned with 
catastrophic failure and isolation to support a maintenance concept than how well the equipment 
is working. Performance monitoring (how well the equipment is working) requires tighter 
tolerances than the classical BIT. 

A problem exists in preserving data integrity where tight tolerances are required. The sensed 
information first exists in electrical (volts, amps, frequency, etc.) or physical (pressure, 
temperature, etc.) form. This information becomes data only when it is interpreted by a person or 
machine. This occurs in the following ways: 

• A person reads an instrument display and interprets the data (the tolerances of the data 
are based on the sensor and display meter); 

• The information is sensed and sent over a transmission line prior to being interpreted as 
above. This sending of information over transmission lines causes degradation; 

• The information is sensed and converted to a digital form (data) prior to sending over the 
transmission line. This tends to preserve the data and ease the problem when the data 
must be sent to a place removed from the hardware. 

In the future, the interpretation of sensed performance data can be used to detect and/or 
predict failures. The prediction of failures permits implementation of preventive maintenance prior 
to catastrophic failures. The classical BIT would be used to isolate failures at the operational 
level. This approach will probably require a standard family of sensors that converts information 
directly to the digital data state. The proposed on-line/off-line interface MIL-STD would contain 
this family of standard sensors along with methods for their application and evaluation. The pri¬ 
mary emphasis would be on the characteristics at the interface. This proposed interface MIL-STD 
or section would be a subset of MIL-STD-XXXX (NOSC Proposed). MIL-STD-XXXX 
subsets would also cover the types of data that are required, by equipment class, to have perfor¬ 
mance monitoring. The projected result of the increase in LSI technology, including self-test, will 
allow for greater self-contained BIT in future military hardware. This increased capacity will allow 
the BIT to perform much of what an off-line tester currently does. 


MIL-STD-1326 

MIDSTEM 326 should be replaced by MIDSTDXXXX (NOSC Proposed). The sections 
on selection of test points and characteristics fit better in MIL-STD-415. The deliverable items 
are Test Requirements Data (TRD), covered in MID STD-2076. Since MIDSTD-1326 was 
drafted in the late 1960s, the interface portion needs updating and will fit better in MIDSTD- 
XXXX (NOSC Proposed). 
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5.0 CONCLUSIONS 


5.1 GENERAL 

Established design disciplines such as performance, maintainability, etc., are adequately 
backed by military specifications and standards. The subject of “testability” as a design discipline 
does not currently exist in military specifications and standards for use in procurement Many 
military documents exist old and recent (MID STD-415, MIL-STD-1326, MIL-STD-2076, etc.), 
that contain testability hardware requirements, but not testability as a design discipline. Also, 
conflict and overlap exist between different documents when different agencies have developed 
their own document to perform an identical function. The specific conclusions (paragraph 5.2) are 
presented by the following testability functions: 

• Management 

• Technical Requirements 

• Design Guide 

• Documentation 

• Validation 

• Interfaces 

5.2 SPECIFIC CONCLUSIONS 

Management * Testability does not currently exist as a design discipline as do the disciplines 
of performance, maintainability, safety, etc. Heretofore, testability has been implemented through 
the maintainability design discipline using MIL-STD-470 as the management document 
However, the satisfaction of contractual maintainability goals (e.g., MTTR) does not in and of 
itself ensure a highly testable design. A similar document for management of testability by an 
R&D contractor is required to enhance testability interests. These management elements are 
available in the W. Keiner document, “A Study of Testability Standardization for Electronic 
Systems and Equipment”, and are proposed as MID STD-ABC, “Testability Program 
Requirements”. 

Technical Requirements - Since testability does not exist as a design discipline in equipment 
specifications and general standards, MIL-STD-490, “Specification Practices”, does not currently 
make provisions for testability as a specification item as it does with maintainability and 
reliability. Also, general standard, MID STD-454, does not have a requirement that is dedicated 
to the subject of testability. Requirement 32, Test Provisions, is associated with the hardware 
requirement; however, it doesn’t address the design discipline of testability. As a point of illustra¬ 
tion, MIDSTD-454 specifies the maintainability requirement by function as follows: 

• MIDSTD-470 Maintainability Program 

• MIDHDBK-472 Maintainability Prediction 
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• MIL-STD-471 Maintainability Verification 

As previously noted, this approach to Testability does not currently exist in released form. MID 
STD-XXX Proposed, W. Keiner, “Testability Acquisition Program Requirements for Electronic 
Equipments and Systems”, covers these topics in a single document 

MID STD-415, Test Provisions for Electronic Systems and Associated Equipment Design 
Criteria, contains a mix of design, management and documentation requirements. 

Design Guide - A design guide is not a contractual document during the procurement of 
hardware. The design guide is supplied to the hardware contractor as an instrument to disseminate 
information. 

The BIT design guide fulfills this information criteria for the classical case. The classical BIT 
was developed to meet the requirements of the maintenance concept Due to this fact BIT is 
limited to GO/NO-GO testing that determines equipment readiness for operation and for failure 
localization. It does not tell how well the equipment is working. 

A design guide for operational monitoring does not currently exist An operational monitoring 
design guide that depicts methods of implementing performance monitoring would support the 
ORMS concept 

The W. Keiner document Design for Testability Course, depicts the methodology for 
implementing testability into the hardware. This course would fulfill the criteria for disseminating 
overall testability information provided that contractor personnel would have access to it An 
overall testability design guide will aid in the education process. 

MIDSTD-1390, Level of Repair, may be used in making testability design trade-off studies. 

MIDSTD-1365, Standard General Purpose Electronic Test Equipment provides a shopping 
list for use by hardware contractors. 

Documentation - The following types of documentation are associated with testability: 

• Performance Specification (MIL-STD-490) 

• Testability Analysis and Report (Proposed) 

• Test Requirements Data (TRD) 

• Testability Demonstration (Proposed) 

• Technical Support Plan (TSP) 

• Test Program Sets (TPS) 

• Technical Repair Standards (TRS) 

• Restoration, Overhaul, and Repair (ROR) 
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Although MIL-STD-490 was discussed under technical requirements, it is also applicable here 
because the performance specifications may become an equipment specification depending on the 
production volume. 

A test analysis and report MID STD does not currently exist in released form. This type of 
analysis, if performed during the hardware development, would influence the design. 

Overlap and conflict exists between the following Test Requirements Data (TRD) MIL-STDs: 

• MIL-STD-1519 

• MIDSTD-1345 

• MID STD-2076 

MID STD-2076 is the more current of the three documents. This MIDSTD pertains to the use of 
ATE, ATLAS and Test Program Sets (TPS). 

The callout for CDRL items and DIDs pertaining to testability demonstration plans, 
procedures and reports does not currently exist. A testability demonstration MIL-STD similar to 
the maintainability demonstration, MIDSTD-471, does r.ot exist in released form. 

The technical support plan (TSP) document, MIDT-24309, does not contain reference to the 
Test Requirements Data (TRD) as specified in MIDSTD-1519, MIDSTD-1345 or MID STD- 
2076. 

The test program sets (TPS) document, MID STD-2077, being slanted towards avionics, is 
not sufficiently universal for use outside NAVAIR. 

The following technical repair standard (TRS) documents were reviewed as part of this study: 

• MIDSTD-2111, Technical Repair Standards (4G repairables), Preparation of 

• MIDT-24255, Technical Repair Standards (For Submarines Only) 

• MIDT-24424, Technical/Maintenance Overhaul and Repair Standards 

These documents are specialized and perform different functions. MIDSTD-2111 is for depot 
repair. MIDT-24255 is for the overhaul of submarines and MIDT-24424 is for the overhaul of 
ships. MIDSTD-2111 does not make specific provisions for ATE and TPS. MIDSTD-3110, 
Restoration, Overhaul, and Repair of Electronic Equipment, is listed under documentation 
because testing is a part of it In the tier of documents, MIL-STD-3110 uses TRS which are 
covered in MIDSTD-2111, MIDT-24255, and MIDT-24424. 

The general purpose electronic test equipment (GPETE) is shown in Figure 4, Testability 
Related Data Flow, for those cases where ATE is not used. The GPETE data are specified in the 
TRD and used in the TRS and ROR documents. The TRS/ROR documents utilize the TRD, 
GPETE and TPS data. 
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Validation - MIL-STD-471 places its priority on how long test equipment must operate. This 
opposes the more important testability aspects of how effectively the test equipment works. The 
controlling factor is the Mean Time To Repair (MTTR) parameter. A document does not exist in 
released form to control the demonstration of the testability of the hardware. 

Interfaces - Many documents have been developed in an attempt to standardize physical, 
performance and contractual interfaces. Because of the varied requirements by different users, a 
single interface will not satisfy everyone. A limited shopping list of interfaces is the logical 
solution. DOD-STD-1399, being a general standard, is the logical place to put the shopping list 

A MIL-Document that deals with the concept of performance monitoring does not currently 
exist in released form. The existing testability related MIDdocuments, with the exception of MILr 
STD-1326, deal with satisfying the maintenance concept The maintenance concept deals with 
how the equipment is operating as well as with isolating failures. It does not address how well it is 
working. MIL-STD-1326 does not meet the current state of the art in testability technology. 

MIL-STD-XXXX (NOSC Proposed), “System Level Performance Monitoring, General 
Requirements”, addresses the subject of performance monitoring. A methodology for developing a 
set of sections or MID STDs by classes of shipboard hardware for performance monitoring is 
presented in MIDSTD-XXXX. Performance monitoring requires tighter tolerances than the 
testing associated with meeting the maintenance concept This will require a family of int 'face 
devices that transfers the physical (pressure, temperature, etc.) and electrical performance ..olts, 
frequency, etc.) parameters directly to the information state (digital). An On-Line/Off-Line 
interface document does not currently exist to document and specify this family of interface 
devices. The projected increase in LSI technology, including self-test, will permit greater self- 
contained BIT in future hardware. This increased capability will allow the BIT to perform much 
of what an off-line tester currently does. This future BIT capability will be part of the family of 
interface devices. 


6.0 RECOMMENDATIONS 


6.1 GENERAL 

The establishment of testability as a design discipline will require generation of a family of 
testability documents that can be applied when procuring hardware. This will require: 

• Generation of new military documents; 

• Cancellation of some military documents; 

• Modification of some military documents. 

The recommended hierarchy of testability documents relative to the procurement of hardware 
is depicted in Figure 5. The following footnotes in Figure 5 apply as follows: 

NOTES: 1. Modify existing document - Change the existing document to avoid conflict 
with other documents and/or to specify testability requirements. 


- 33 - 



I 


Figure 5. Recommended Hierarchy of Testability E 

























procurement Office 


ements 


i 


Statement 
of Work (SOW) 


I 


Tailor MIL-STDs 
& Specifications 


I 


Tasks 


Deliverables 



I 


MIL-STD-XXXX (NOSC) (2) 

System Level Performance 
Monitoring, General Requirements 


Radar 

Systems 


(2) 


I 


E 



Communication 


Combat 


Hull 



Systems 


Systems 


Machinery 



(2) 


(2) 


(2) 



r- 


MIL-STD-X (2) 
On-Line/ 
Off-Line 
Interfaces 



Design Guides 

Testability (2) 

Performance 
Monitoring (2) 

BIT 


Documentation 


- MIL-STD-2076 (1) 

- MIL-STD-2077 (1) 

- MIL-T-24309 (1) 

- MIL-STD-2111 (1) 

- MIL-T-24255 

- MIL-T-24424 

- MIL-STD-3110 

- MIL-STD-1521 (1) 


IM Hierarchy of Testability Documents 


i 




jCBCraa PACK BUMMIOV numn 


- 35 - 













2. Develop New Document - A new MIDdocument is required to specify 
testability requirements and/or complete the family of testability documents 

The generation, cancellation and/or modification of each applicable military document is 
discussed in paragraph 6.2 under the following testability functions: 

• Management 

• Technical Requirements 

• Design Guide 

• Documentation 

• Validation 

• Interfaces 


6.2 SPECIFIC RECOMMENDATIONS 

Management - A Testability Program document (MIL-STD-ABC, Proposed), similar to 
MIL-STD-470 for maintainability, should be developed. This testability program document would 
be derived from the program elements contained in the W. Keiner work, “A Study of Testability 
Standardization for Electronic Systems and Equipment”. 

Technical Requirements • The following will be required to establish testability as a design 
discipline and develop a family of MIL-documents that passes testability requirements to allied 
documents: 

• Modify MIDSTD-490 to include testability requirements; 

• Modify MIDE-16400, MIDT-28800, etc., to include testability requirements, and to 
specifically include a reference to MID STD-45 4; 

• Modify MIDSTD-454 to include requirement sections for testability and interfaces. The 
testability requirement would call out the following MID STDs proposed: 

- Testability Program Requirements (MIDSTD-ABC) 

- Testability Analysis and Report (MIDSTD-GHI) 

- Testability Demonstration (MIDSTD-DEF) 

NOTE: The three-document approach is in parallel with that approach used by 

maintainability. The proposed interface section for MIDSTD-454 would reference 
MIDSTD-1657 and possibly MIDSTD-1399. 


• Modify MIL- STD-4 IS to contain only design and hardware requirements. The manage¬ 
ment information should be in the proposed MID STD- ABC “Testability Program 
Requirements". The documentation requirements should be in the TRD document; 

• Technical requirements as well as the test point placement and compatibility aspects of 
testability must be stated for the build-in test Whether or not their statement is best 
promulgated under a single document (modified MID STD-415) or under two documents 
is unresolved at this time. 

Design Guide - The following testability design guides should be developed to aid in the 
testability educational process: 

• Performance Monitoring Testability Design Guide which presents methods and examples 
of performance monitoring in hardware; 

• An overall Design for Testability Guide should be developed. The conversion of the 
W. Keiner document, “Design for Testability Course”, to a written test will fulfill this 
need. 

Documentation - The following modifications are needed in the testability documentation 
area: 

• Modify MID STD-490 to include testability requirements; 

• Develop a test analysis and report (MIDSTD-GHI) utilizing data from the W. Keiner 
report, “A Study of Testability Standardization of Electronic Systems and Equipment"; 

• Develop a test requirements data (TRD) MID STD (TBD) utilizing MID STD-2076 as a 
base. This would replace MIDSTD-1519 and MIDSTD-1345; 

• Modify MIDT-24309 to reference test requirements data (TRD) as specified in 
MIDSTD-1519, MIDSTD-1345 or MIDSTD-2076; 

• Modify MIDSTD-2077 to cover more than avionics; 

• Modify MIDSTD-2111 to cover ATE and TPS; 

• Develop CDRL Item and supporting DIDs; 

• Modify MIDSTD-1521 to include testability in checklists for the design reviews (SRR, 
SDR, PDR, CDR) and audits (FCA, PCA). 

Validation - Develop a testability demonstration MIDSTD-DEF similar to the maintain¬ 
ability demonstration document, MIL-STD-471. Utilize data from the W. Keiner report, “A 
Study of Testability Standardization for Electronic Systems and Equipment”. This will include 
the testability demonstration plan, procedure and report 

Interfaces - The following recommendations are submitted to standardize interface 
requirements. 
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• Utilize DOD-STD-1399 as the general standard for interfaces. 

• Cancel MIDSTD-1326. The section on test points and characteristics should be in 
MIL-STD-415. The deliverable items are Test Requirement Data (TRD) and should be in 
MIL-STD-2076. The remainder is obsolete. 

• Develop MIL-STD-XXXX (NOSC Proposed) to replace MID STD-1326. 

• Develop an On-Line/Off-Line Interface document that specifies a family of interface 
devices that convert physical and electrical performance parameters to an information 
state (digital). 

Implementation - What has been presented herein in a roadmap for the development of the 
MIL-documents required to facilitate making testability happen in military equipment The next 
logical question is, “which roads do we take and when?” The functional partitioning utilized in 
this study is also useful for dividing the work that should follow. Each functional area can be 
approached separately (with due attention to interactions) for the definition of work to be done 
and for the generation of documents. 

The following program is recommended. Each functional area should be investigated more 
deeply to turn this generalized plan into a more detailed implementation plan. These plans should 
be sufficiently detailed to serve as a basis to effectively estimate the time and effort required to 
achieve the final goal. Specifically, they should answer the following questions: 

• What effort is required to develop all the documents required? 

• What is the total time required and interim milestones? 

• What supporting documentation (from other areas) is required for the interim milestones? 

• What supporting technology development/investigation is required? 

• Can an overall time-phased implementation provide meaningful interim products? 

• What existing work is available? 

• What can a “trial run” accomplish? 
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